Capturing and Processing of Images Including Occlusions Captured by Heterogeneous Camera Arrays

ABSTRACT

Systems and methods for implementing array cameras configured to perform super-resolution processing to generate higher resolution super-resolved images using a plurality of captured images and lens stack arrays that can be utilized in array cameras are disclosed. Lens stack arrays in accordance with many embodiments of the invention include lens elements formed on substrates separated by spacers, where the lens elements, substrates and spacers are configured to form a plurality of optical channels, at least one aperture located within each optical channel, at least one spectral filter located within each optical channel, where each spectral filter is configured to pass a specific spectral band of light, and light blocking materials located within the lens stack array to optically isolate the optical channels.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/475,481 filed Sep. 2, 2014, which application is a continuation ofU.S. Pat. No. 8,861,089, issued Oct. 14, 2014, which application is acontinuation of U.S. Pat. No. 8,514,491, issued Aug. 20, 2013, whichapplication claimed priority to U.S. Provisional Patent Application No.61/281,662 filed Nov. 20, 2009 and U.S. Provisional Patent ApplicationNo. 61/263,339 filed Nov. 20, 2009, the disclosures of which isincorporated by reference herein in its entirety.

FIELD OF THE INVENTION

The present invention is related to an image sensor including aplurality of heterogeneous imagers, more specifically to an image sensorwith a plurality of wafer-level imagers having custom filters, sensorsand optics of varying configurations.

BACKGROUND

Image sensors are used in cameras and other imaging devices to captureimages. In a typical imaging device, light enters through an opening(aperture) at one end of the imaging device and is directed to an imagesensor by an optical element such as a lens. In most imaging devices,one or more layers of optical elements are placed between the apertureand the image sensor to focus light onto the image sensor. The imagesensor consists of pixels that generate signals upon receiving light viathe optical element. Commonly used image sensors include CCD(charge-coupled device) image sensors and CMOS (complementarymetal-oxide-semiconductor) sensors.

Filters are often employed in the image sensor to selectively transmitlights of certain wavelengths onto pixels. A Bayer filter mosaic isoften formed on the image sensor. The Bayer filter is a color filterarray that arranges one of the RGB color filters on each of the colorpixels. The Bayer filter pattern includes 50% green filters, 25% redfilters and 25% blue filters. Since each pixel generates a signalrepresenting strength of a color component in the light and not the fullrange of colors, demosaicing is performed to interpolate a set of red,green and blue values for each image pixel.

The image sensors are subject to various performance constraints. Theperformance constraints for the image sensors include, among others,dynamic range, signal to noise (SNR) ratio and low light sensitivity.The dynamic range is defined as the ratio of the maximum possible signalthat can be captured by a pixel to the total noise signal. Typically,the well capacity of an image sensor limits the maximum possible signalthat can be captured by the image sensor. The maximum possible signal inturn is dependent on the strength of the incident illumination and theduration of exposure (e.g., integration time, and shutter width). Thedynamic range can be expressed as a dimensionless quantity in decibels(dB) as:

$\begin{matrix}{{DR} = {\frac{{full}{\mspace{11mu} \;}{well}\mspace{14mu} {capacity}}{{RMS}\mspace{14mu} {noise}}\mspace{14mu} \ldots \mspace{14mu} {equation}}} & (1)\end{matrix}$

Typically, the noise level in the captured image influences the floor ofthe dynamic range. Thus, for an 8 bit image, the best case would be 48dB assuming the RMS noise level is 1 bit. In reality, however, the RMSnoise levels are higher than 1 bit, and this further reduces the dynamicrange.

The signal to noise ratio (SNR) of a captured image is, to a greatextent, a measure of image quality. In general, as more light iscaptured by the pixel, the higher the SNR. The SNR of a captured imageis usually related to the light gathering capability of the pixel.

Generally, Bayer filter sensors have low light sensitivity. At low lightlevels, each pixel's light gathering capability is constrained by thelow signal levels incident upon each pixel. In addition, the colorfilters over the pixel further constrain the signal reaching the pixel.IR (Infrared) filters also reduce the photo-response from near-IRsignals, which can carry valuable information.

These performance constraints of image sensors are greatly magnified incameras designed for mobile systems due to the nature of designconstraints. Pixels for mobile cameras are typically much smaller thanthe pixels of digital still cameras (DSC). Due to limits in lightgathering ability, reduced SNR, limits in the dynamic range, and reducedsensitivity to low light scenes, the cameras in mobile cameras show poorperformance.

SUMMARY

Lens stack arrays that can be utilized in camera arrays in accordancewith embodiments of the invention are disclosed. In many embodiments,lens elements are provided that can direct and focus light onto theimagers of a camera array. The lens elements form lens stacks thatcreate optical channels, and each lens stack focuses light onto oneimager. Because each lens element is associated with one imager, eachlens element may be designed and configured for a narrow light spectrum.Further, the thickness of the lens element may be reduced, decreasingthe overall thickness of the camera array. In such an embodiment, thelens elements may be made using any suitable fabrication technique, suchas, for example using wafer level optics (WLO) technology, injectionmolding, and/or glass molding.

A lens stack array in accordance with another embodiment of theinvention includes lens elements formed on substrates separated byspacers, where the lens elements, substrates and spacers are configuredto form a plurality of optical channels, at least one aperture locatedwithin each optical channel, at least one spectral filter located withineach optical channel, where each spectral filter is configured to pass aspecific spectral band of light, and light blocking materials locatedwithin the lens stack array to optically isolate the optical channels.

In a further embodiment, the light blocking materials are selected fromthe group consisting of opaque materials, reflective materials andcombinations thereof.

In another embodiment, spectral filters that pass different spectralbands are provided within at least two of the imagers.

In a still further embodiment, each spectral filter is selected from thegroup consisting of an organic color filter, an absorptive material, adielectric coating, an interference filter, a multilayer coating, andcombinations thereof.

Still another embodiment also includes at least one polarizing filterlocated within each optical channel.

In a yet further embodiment, the construction of each optical channeldiffers based upon the specific spectral band of light passed by thespectral filter within the imager so that chromatic aberrations arereduced.

In yet another embodiment, the prescription of at least one surface of alens within each optical channel is a function of the specific spectralband of light passed by the spectral filter within the optical channel.

In a further embodiment again, the back focal lengths of each opticalchannel in the lens stack array are the same irrespective of thespectral band of light passed by the spectral filter within the opticalchannel.

In another embodiment again, a combination of high and low Abbe numbermaterials is used in the construction of each of the lens elements in anoptical channel to reduce chromatic aberrations.

A further additional embodiment also includes at least one aperture stoplocated within each optical channel. In addition, each spectral filteris located within each optical channel so that the spectral filter isproximate the aperture stop.

In another additional embodiment, each aperture stop is formed by alight blocking material selected from the group consisting of metalmaterials, oxide materials, black particle filled photoresists andcombinations thereof.

In a still yet further embodiment, the at least one lens surface of alens in each optical channel differs based upon the specific spectralband of light passed by the spectral filter within the optical channel,and each lens surface is selected from the group consisting ofdiffractive, Fresnel, refractive and combinations thereof.

In still yet another embodiment, the radii of curvature of the lenssurfaces differ based upon the specific spectral band of light passed bythe spectral filter within the optical channel.

In a still further embodiment again, each of the optical channels havethe same back focal length.

In still another embodiment again, at least one of the lens elements inan optical channel is a negative lens element, and the negative lenselement is proximate the image formed by the optical channel.

In a still further additional embodiment, at least two of the opticalchannels have different focal lengths.

Still another additional embodiment also includes a mechanical zoommechanism within an optical channel configured to smoothly transitionbetween different fields of view.

In a yet further embodiment again, the lens stack array comprises an N×Marray of optical channels, where at least one of N and M is greater than2.

In yet another embodiment again, the lens stack array is fabricatedusing techniques consisting of wafer level optics techniques, injectionmolding, glass molding and combinations thereof.

In a yet further additional embodiment, the light blocking materialslocated within the lens stack array to optically separate the opticalchannels comprise at least two opaque surfaces located on a substratewithin an optical channel, where the two opaque surfaces have openingsarranged in axial alignment with the optical channel.

In yet another additional embodiment, the light blocking materialslocated within the lens stack array to optically separate the opticalchannels comprise opaque walls disposed at the boundaries between theoptical channels.

In a still further additional embodiment again, the opaque walls arecavities between the optical channels filled with light blockingmaterial.

In still another additional embodiment again, a plurality of the spacersare constructed from light blocking materials and are located within thelens stack array to optically separate the optical channels.

In a yet further additional embodiment again, a plurality of the spacersare coated in light blocking materials and are located within the lensstack array to optically isolate the optical channels.

The features and advantages described in the specification are not allinclusive and, in particular, many additional features and advantageswill be apparent to one of ordinary skill in the art in view of thedrawings, specification, and claims. Moreover, it should be noted thatthe language used in the specification has been principally selected forreadability and instructional purposes, and may not have been selectedto delineate or circumscribe the inventive subject matter.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a plan view of a camera array with a plurality of imagers,according to one embodiment.

FIG. 2A is a perspective view of a camera array with lens elements,according to one embodiment.

FIG. 2B is a cross-sectional view of a camera array, according to oneembodiment.

FIG. 2C is a cross-sectional view of a camera array with opticalcrosstalk suppression, according to one embodiment.

FIG. 2D is a cross-sectional view of a camera array with opticalcrosstalk suppression, according to a second embodiment.

FIG. 2E is a cross-sectional view of a camera array incorporating opaquespacers to provide optical crosstalk suppression, according to a furtherembodiment.

FIG. 2F is a cross-sectional view of a camera array incorporatingspacers coated with opaque material to provide crosstalk suppression,according to another embodiment.

FIGS. 3A and 3B are sectional diagrams illustrating changes in theheights of lens elements depending on changes in the dimensions ofimagers, according to one embodiment.

FIG. 3C is a diagram illustrating chief ray angles varying depending ondiffering dimensions of the lens elements.

FIG. 3D is a cross-sectional view of a camera array with fieldflattening, according to one embodiment.

FIG. 4 is a functional block diagram for an imaging device, according toone embodiment.

FIG. 5 is a functional block diagram of an image processing pipelinemodule, according to one embodiment.

FIGS. 6A through 6F are plan views of camera arrays having differentlayouts of heterogeneous imagers, according to embodiments.

FIG. 6G is a diagram conceptually illustrating the manner in whichsampling diversity can depend upon object distance.

FIG. 6H is a is a cross sectional view of pixels of an imager inaccordance with an embodiment of the invention

FIG. 6I is a diagram conceptually illustrating occlusion zones createdwhen Red and Blue imagers are not symmetrically distributed about thecentral access of a camera array.

FIG. 6J is a diagram conceptually illustrating the manner in which theocclusion zones illustrated in FIG. 6I are eliminated by distributingRed and Blue imagers symmetrically about the central access of a cameraarray.

FIG. 7 is a flowchart illustrating a process of generating an enhancedimage from lower resolution images captured by a plurality of imagers,according to one embodiment.

FIG. 7A is a flow chart illustrating a process for constructing anormalization plane during calibration in accordance with an embodimentof the invention.

FIG. 7B conceptually illustrates the process for constructing anormalized plane during calibration in accordance with an embodiment ofthe invention illustrated in FIG. 7A.

FIG. 8A is a cross-sectional view of a camera array with optical zoom,according to one embodiment.

FIG. 8B is a cross-sectional view of a camera array with optical zoom,according to a second embodiment.

FIG. 8C is a cross-sectional view of a camera array with imagers havingdifferent fields-of-view, according to a further embodiment.

DETAILED DESCRIPTION

Embodiments of the invention are now described with reference to thefigures where like reference numbers indicate identical or functionallysimilar elements. Also in the figures, the left most digits of eachreference number corresponds to the figure in which the reference numberis first used.

Many embodiments relate to using a distributed approach to capturingimages using a plurality of imagers of different imagingcharacteristics. Each imager may be configured in such a manner thateach imager captures an image that is shifted by a sub-pixel amount withrespect to the image captured by other imagers having similar imagingcharacteristics. Each imager may also include separate optics withdifferent filters and operate with different operating parameters (e.g.,exposure time). Distinct images generated by the imagers are processedto obtain an enhanced image. In many embodiments, the separate opticsincorporated into each imager are implemented using a lens stack array.The lens stack array can include one or more optical elements fabricatedusing wafer level optics (WLO) technology.

A sensor element or pixel refers to an individual light sensing elementin an imager. The light sensing element can be, but is not limited to,traditional CIS (CMOS Image Sensor), CCD (charge-coupled device), highdynamic range pixel, multispectral pixel and various alternativesthereof.

A sensor refers to a two dimensional array of pixels used to capture animage formed on the sensor by the optics of the imager. The sensorelements of each sensor have similar physical properties and receivelight through the same optical component. Further, the sensor elementsin the each sensor may be associated with the same color filter.

A camera array refers to a collection of imagers designed to function asa unitary component. The camera array may be fabricated on a single chipfor mounting or installing in various devices.

An array of camera arrays refers to an aggregation of two or more cameraarrays. Two or more camera arrays may operate in conjunction to provideextended functionality over a single camera array, such as, for example,stereo resolution.

Image characteristics of an imager refer to any characteristics orparameters of the imager associated with capturing of images. Theimaging characteristics may include, among others, the size of theimager, the type of pixels included in the imager, the shape of theimager, filters associated with the imager, the exposure time of theimager, aperture size associated with the imager, the configuration ofthe optical element associated with the imager (such as the number ofelements, the shapes, profiles and sizes of the lens surfaces, includingthe radii of curvature, aspheric coefficients, focal lengths and FOVs ofthe objectives, color correction, F/#s, etc.), the gain of the imager,the resolution of the imager, and operational timing of the imager.

Structure of Camera Array

FIG. 1 is a plan view of a camera array 100 with imagers 1A through NM,according to one embodiment. The camera array 100 is fabricated on asemiconductor chip to include a plurality of imagers 1A through NM. Eachof the imagers 1A through NM may include a plurality of pixels (e.g.,0.32 Mega pixels). In one embodiment, the imagers 1A through NM arearranged into a grid format as illustrated in FIG. 1. In otherembodiments, the imagers are arranged in a non-grid format. For example,the imagers may be arranged in a circular pattern, zigzagged pattern orscattered pattern or an irregular pattern including sub-pixel offsets.

The camera array may include two or more types of heterogeneous imagers,each imager including two or more sensor elements or pixels. Each one ofthe imagers may have different imaging characteristics. Alternatively,there may be two or more different types of imagers where the same typeof imager shares the same imaging characteristics.

In one embodiment, each imager 1A through NM has its own filter and/oroptical element (e.g., lens). Specifically, each of the imagers 1Athrough NM or a group of imagers may be associated with spectral colorfilters to receive certain wavelengths of light. Example filters includea traditional filter used in the Bayer pattern (R, G, B or theircomplements C, M, Y), an IR-cut filter, a near-IR filter, a polarizingfilter, and a custom filter to suit the needs of hyper-spectral imaging.Some imagers may have no filter to allow reception of both the entirevisible spectra and near-IR, which increases the imager'ssignal-to-noise ratio. The number of distinct filters may be as large asthe number of imagers in the camera array. Further, each of the imagers1A through NM or a group of imagers may receive light through lenseshaving different optical characteristics (e.g., focal lengths) orapertures of different sizes.

In one embodiment, the camera array includes other related circuitry.The other circuitry may include, among others, circuitry to controlimaging parameters and sensors to sense physical parameters. The controlcircuitry may control imaging parameters such as exposure times, gain,and black level offset. The sensor may include dark pixels to estimatedark current at the operating temperature. The dark current may bemeasured for on-the-fly compensation for any thermal creep that thesubstrate may suffer from. Alternatively, compensation of thermaleffects associated with the optics, e.g., because of changes inrefractive index of the lens material, may be accomplished bycalibrating the PSF for different temperatures.

In one embodiment, the circuit for controlling imaging parameters maytrigger each imager independently or in a synchronized manner. The startof the exposure periods for the various imagers in the camera array(analogous to opening a shutter) may be staggered in an overlappingmanner so that the scenes are sampled sequentially while having severalimagers being exposed to light at the same time. In a conventional videocamera sampling a scene at N exposures per second, the exposure time persample is limited to 1/N seconds. With a plurality of imagers, there isno such limit to the exposure time per sample because multiple imagersmay be operated to capture images in a staggered manner.

Each imager can be operated independently. Entire or most operationsassociated with each individual imager may be individualized. In oneembodiment, a master setting is programmed and deviation (i.e., offsetor gain) from such master setting is configured for each imager. Thedeviations may reflect functions such as high dynamic range, gainsettings, integration time settings, digital processing settings orcombinations thereof. These deviations can be specified at a low level(e.g., deviation in the gain) or at a higher level (e.g., difference inthe ISO number, which is then automatically translated to deltas forgain, integration time, or otherwise as specified by context/mastercontrol registers) for the particular camera array. By setting themaster values and deviations from the master values, higher levels ofcontrol abstraction can be achieved to facilitate a simpler programmingmodel for many operations. In one embodiment, the parameters for theimagers are arbitrarily fixed for a target application. In anotherembodiment, the parameters are configured to allow a high degree offlexibility and programmability.

In one embodiment, the camera array is designed as a drop-in replacementfor existing camera image sensors used in cell phones and other mobiledevices. For this purpose, the camera array may be designed to bephysically compatible with conventional image sensors of approximatelythe same resolution although the achieved resolution of the camera arraymay exceed conventional image sensors in many photographic situations.Taking advantage of the increased performance, the camera array inaccordance with embodiments of the invention may include fewer pixels toobtain equal or better quality images compared to conventional imagesensors. Alternatively, the size of the pixels in the imager may bereduced compared to pixels in conventional image sensors while achievingcomparable results.

In order to match the raw pixel count of a conventional image sensorwithout increasing silicon area, the logic overhead for the individualimagers is preferably constrained in the silicon area. In oneembodiment, much of the pixel control logic is a single collection offunctions common to all or most of the imagers with a smaller set offunctions applicable to each imager. In this embodiment, theconventional external interface for the imager may be used because thedata output does not increase significantly for the imagers.

In one embodiment, the camera array including the imagers replaces aconventional image sensor of M megapixels. The camera array includes N×Nimagers, each sensor including pixels of

$\frac{M}{N^{2}}.$

Each imager in the camera array also has the same aspect ratio as theconventional image sensor being replaced. Table 1 lists exampleconfigurations of camera arrays according to the present inventionreplacing conventional image sensor.

TABLE 1 Conventional Camera array Including Imagers Image Sensor No. ofNo. of Super- Total Effective Total Horizontal Vertical ImagerResolution Effective Mpixels Resolution Mpixels Imagers Imagers MpixelsFactor Resolution 8 3.2 8 5 5 0.32 3.2 3.2 8 4 4 0.50 2.6 3.2 8 3 3 0.891.9 3.2 5 2.0 5 5 5 0.20 3.2 2.0 5 4 4 0.31 2.6 2.0 5 3 3 0.56 1.9 2.0 31.2 3 5 5 0.12 3.2 1.2 3 4 4 0.19 2.6 1.2 3 3 3 0.33 1.9 1.2

The Super-Resolution Factors in Table 1 are estimates and the EffectiveResolution values may differ based on the actual Super-Resolutionfactors achieved by processing.

The number of imagers in the camera array may be determined based on,among other factors, (i) resolution, (ii) parallax, (iii) sensitivity,and (iv) dynamic range. A first factor for the size of imager is theresolution. From a resolution point of view, the preferred number of theimagers ranges from 2×2 to 6×6 because an array size of larger than 6×6is likely to destroy frequency information that cannot be recreated bythe super-resolution process. For example, 8 Megapixel resolution with2×2 imager will require each imager to have 2 Megapixels. Similarly, 8Megapixel resolution with a 5×5 array will require each imager to have0.32 Megapixels. In many embodiments, the number of imagers in the arrayis determined based upon the requirements of a specific application.

A second factor that may constrain the number of imagers is the issue ofparallax and occlusion. With respect to an object captured in an image,the portion of the background scene that is occluded from the view ofthe imager can be referred to as the “occlusion set.” When two imagerscapture the object from two different locations, the occlusion set ofeach imager is different. Hence, there may be scene pixels captured byone imager but not the other. To resolve this issue of occlusion, it isdesirable to include a certain minimal set of imagers for a given typeof imager and to distribute the imagers symmetrically around the centralaxis of the camera array.

A third factor that may put a lower bound on the number of imagers isthe issue of sensitivity in low light conditions. To improve low lightsensitivity, imagers for detecting near-IR spectrum may be needed. Thenumber of imagers in the camera array may need to be increased toaccommodate such near-IR imagers.

A fourth factor in determining the size of the imager is dynamic range.To provide dynamic range in the camera array, it is advantageous toprovide several imagers of the same filter type (chroma or luma). Eachimager of the same filter type may then be operated with differentexposures simultaneously. The images captured with different exposuresmay be processed to generate a high dynamic range image.

Based on these factors, the preferred number of imagers is 2×2 to 6×6.4×4 and 5×5 configurations are more preferable than 2×2 and 3×3configurations because the former are likely to provide sufficientnumber of imagers to resolve occlusion issues, increase sensitivity andincrease the dynamic range. In addition, rectangular arrays can also bepreferred. At the same time, the computational load required to recoverresolution from these array sizes will be modest in comparison to thatrequired in the 6×6 array. Arrays larger than 6×6 may, however, be usedto provide additional features such as optical zooming and multispectralimaging. Although only square imagers are described here, as will bediscussed in greater detail later, such imagers may have different x-and y-dimensions.

Another consideration is the number of imagers dedicated to lumasampling. By ensuring that the imagers in the array dedicated to near-IRsampling do not reduce the achieved resolution, the information from thenear-IR images is added to the resolution captured by the luma imagers.For this purpose, at least 50% of the imagers may be used for samplingthe luma and/or near-IR spectra. In one embodiment with 4×4 imagers, 4imagers samples luma, 4 imagers samples near-IR, and the remaining 8imagers sample two chroma (Red and Blue). In another embodiment with 5×5imagers, 9 imagers sample luma, 8 imagers sample near-IR, and theremaining 8 imagers sample two chroma (Red and Blue). Further, theimagers with these filters may be arranged symmetrically within thecamera array to address occlusion due to parallax. In a furtherembodiment with 5×5 imager, 17 imagers sample luma, 4 images sample Red,and 4 imagers sample Blue.

In one embodiment, the imagers in the camera array are spatiallyseparated from each other by a predetermined distance. By increasing thespatial separation, the parallax between the images captured by theimagers may be increased. The increased parallax is advantageous wheremore accurate distance information is important. Separation between twoimagers may also be increased to approximate the separation of a pair ofhuman eyes. By approximating the separation of human eyes, a realisticstereoscopic 3D image may be provided to present the resulting image onan appropriate 3D display device.

In one embodiment, multiple camera arrays are provided at differentlocations on a device to overcome space constraints. One camera arraymay be designed to fit within a restricted space while another cameraarray may be placed in another restricted space of the device. Forexample, if a total of 20 imagers are required but the available spaceallows only a camera array of 1×10 imagers to be provided on either sideof a device, two camera arrays each including 10 imagers may be placedon available space at both sides of the device. Each camera array may befabricated on a substrate and be secured to a motherboard or other partsof a device. In addition, such imagers do not have to be homogenous insize, and may have different x- and y-dimensions. The images collectedfrom multiple camera arrays may be processed to generate images ofdesired resolution and performance.

A design for a single imager may be applied to different camera arrayseach including other types of imagers. Other variables in the cameraarray such as spatial distances, color filters and combination with thesame or other sensors may be modified to produce a camera array withdiffering imaging characteristics. In this way, a diverse mix of cameraarrays may be produced while maintaining the benefits from economies ofscale.

Wafer Level Optics Integration

In one embodiment, the camera array employs wafer level optics (WLO)technology. Although in many embodiments, similar optical channels canbe constructed using any of a variety of techniques including but notlimited to injection molding, glass molding, and/or combinations ofthese techniques with other techniques including WLO techniques. WLOitself is a technology that encompasses a number of processes,including, for example, molding optics (such as arrays of lens modulesand arrays of those lens arrays) on glass wafers, stacking of thosewafers (including wafers having lenses replicated on either side of thesubstrate) with appropriate spacers, at either a wafer or die-level,followed by packaging of the optics directly with the imager into amonolithic integrated module.

The WLO procedure may involve, among other procedures, using adiamond-turned mold to create each polymer lens element on a glasssubstrate. More specifically, the process chain in WLO generallyincludes producing a diamond turned lens master (both on an individualand array level), then producing a negative mold for replication of thatmaster (also called a stamp or tool), and then finally forming a polymerreplica on a glass substrate, which has been structured with appropriatesupporting optical elements, such as, for example, apertures, lightblocking materials, filters, etc.

FIG. 2A is a perspective view of a camera array assembly 200 with waferlevel optics 210 and a sensor array 230, according to one embodiment.The wafer level optics 210 includes a plurality of lens elements 220,each lens element 220 covering one of twenty-five imagers 240 in thesensor array 230. Note that the camera array assembly 200 has an arrayof smaller lens elements occupy much less space compared to a singlelarge lens covering the entire sensor array 230. It should also be notedthat each of the lenses may be of a different type. For example, eachsubstrate level may contain lenses that are diffractive, refractive,Fresnel, or a combination thereof. It should be further noted that inthe context of the camera array, that a lens element 220 may compriseone or multiple separate optical lens elements axially arranged withrespect to each another. Finally, it should be noted that, for most lensmaterials there will be a thermal induced variance in the refractiveindex of the material, which must be corrected to obtain good imagequality. A temperature normalization procedure will be described ingreater detail in the sections to follow. FIG. 2B is a sectional view ofa camera array assembly 250, according to one embodiment. The cameraassembly 250 includes a top lens wafer 262, a bottom lens wafer 268, asubstrate 278 with multiple sensors and associated light sensingelements formed thereon and spacers 258, 264 and 270. The camera arrayassembly 250 is packaged within an encapsulation 254. An optional topspacer 258 may be placed between the encapsulation 254 and the top lenswafer 262; however, it is not essential to the construction of thecamera assembly 250. Optical elements 288 are formed on the top lenswafer 262. Although these optical elements 288 are shown as beingidentical in FIG. 2B, it should be understood that different types,sizes, and shapes of elements may be used. A middle spacer 264 is placedbetween the top lens wafer 262 and a bottom lens wafer 268. Another setof optical elements 286 is formed on the bottom lens wafer 268. A bottomspacer 270 is placed between the bottom lens wafer 268 and the substrate278. Through-silicon vias 274 are also provided to paths fortransmitting signal from the imagers. The top lens wafer 262 may bepartially coated with light blocking materials 284 (see discussionbelow) to block of light. The portions of the top lens wafer 262 notcoated with the blocking materials 284 serve as aperture stops throughwhich light passes to the bottom lens wafer 268 and the light sensingelements. Although only a single aperture stop is shown in theembodiment provided in FIG. 2B, it should be understood that additionalaperture stops may be formed from opaque layers disposed on any and allof the substrate faces in the camera assembly to improve stray lightperformance and reduced optical crosstalk. A fuller discussion ofoptical crosstalk suppression is provided below. In addition, althoughthe above embodiment is shown with spacers 258, 264 and 270 the spacerfunction can also be directly implemented by modifying the lensstructures (or substrates) so that the lenses can be directlyinterconnected. In such an embodiment, the lens height can be extended,and the lens glued directly to the upper substrates thereby eliminatingthe need for spacer layers.

In the embodiment of FIG. 2B, filters 282 are formed on the bottom lenswafer 268. Light blocking materials 280 may also be coated on the bottomlens 268 to function as an optical isolator. A light blocking material280 may also be coated on the substrate 278 to protect the sensorelectronics from incident radiation. Spacers 283 can also be placedbetween the bottom lens wafer 268 and the substrate 278 and between thelens wafers 262, 268. In many embodiments, the spacers 283 are similarto the spacers 264 and 270. In a number of embodiments, each layer ofspacers is implemented using a single plate. Although not illustrated inFIG. 2B, many embodiments of the invention also include spacers betweeneach optical channel located on top of the top lens wafer 262 that aresimilar to, or implemented in single layer with, the spacer 258 shown atthe edge of the lens stack array. As is discussed further below thespacers can be constructed from and/or coated in light blockingmaterials to isolate the optical channels formed by the wafer leveloptics. For the purposes of this application, suitable light blockingmaterials may include any opaque material, such as, for example, a metalmaterial like Ti and Cr, or an oxide of these materials like blackchromium (chrome and chrome oxide), or dark silicon, or a black particlefilled photoresist like a black matrix polymer (PSK2000 from BrewerScience). The bottom surface of the substrate is covered with a backsideredistribution layer (“RDL”) and solder balls 276. In one embodiment,the camera array assembly 250 includes 5×5 array of imagers. The cameraarray 250 has a width W of 7.2 mm, and a length of 8.6 mm. Each imagerin the camera array may have a width S of 1.4 mm. The total height t1 ofthe optical components is approximately 1.26 mm and the total height t2the camera array assembly is less than 2 mm. Other heights t1 and t2 arepossible for different lens designs.

Optical Crosstalk Suppression

As discussed above, the camera array assembly 250 is composed ofmultiple imagers, each of which, as shown in FIGS. 2A and 2B, have acorresponding optical pathway or channel that directs light from thescene through the top lens wafer 262, the middle spacer 264, bottom lenswafer 268, the bottom spacer 270 and onto a plurality of light sensingelements that form a sensor 240 disposed on the substrate 278. It isimportant to final image quality that the light impinging on anyparticular sensor come only from its designated optical pathway orchannel. Optical crosstalk can be considered to occur when light that isincident on the top of one imager is also received by light sensingelements of another imager within the array. Any crosstalk betweenoptical channels from, for example, diffraction and/or scattering oflight from elements within the camera, can introduce artifacts in theimage data. In particular, crosstalk between optical channels means thatan imager will sense the flux from a source on the imager that isinconsistent with the reconstructed position of the image of thatdetector and the position of the image. This results in both a loss ofimage data, and the introduction of overlapping noise that cannot bedistinguished from real image data. Accordingly, all optical channels ofthe camera array should be optically isolated so that a ray of lightfrom one lens or optical channel cannot cross from one optical channelto the other. In one embodiment, shown in FIG. 2C, opaque spacers 281 orvertical opaque walls 282 are disposed between each of the opticalchannels 284. Although opaque spacers do provide a level of opticalcrosstalk suppression, vertical opaque walls are preferable because insuch an embodiment both the space between substrates and the relevantsections of the substrates themselves are rendered non-transparent.

The optical crosstalk suppressing vertical opaque walls may be madeusing any suitable technique that provides for the introduction of anopaque surface or material between the optical channels 284 of thecamera array assembly 286. In one embodiment, the vertical opaque wallsare formed by fully or partially introducing grooves into the lens stack288 of the camera array assembly 286. It is preferable not to cut thegrooves fully through the lens stack to preserve the mechanicalintegrity of the camera array assembly. Such grooves may be introducedby any suitable technique, such as, for example, by dicing into thefront or backside of the lens array stack 286 using a wafer dicer(disk/blade), or by laser cutting, or water-jet cutting. Once thegrooves are formed, they are filled with a light blocking material.Alternatively, the inner side walls of the grooves may be coated with alight blocking material and the remainder of the groove filled withanother material with low shrinkage properties. As discussed above, alight blocking material is any opaque material, such as, for example, ametal material, a metal oxide, dark silicon, or a black particle filledphotoresist like a black matrix polymer.

In another embodiment, shown schematically in FIG. 2D, optical crosstalksuppression is achieved by creating a virtual opaque wall formed by aseries of stacked apertures. In this embodiment, a series of aperturestops are formed on the various substrate levels 290 of the camera arrayassembly 292 by coating the substrates with opaque layers 294 providedwith a narrow opening or aperture 296. If enough of these apertures areformed, it is possible to mimic the optical isolation provided by avertical opaque wall. In such a system, a vertical wall would be themathematical limit of stacking apertures one on top of each other.Preferably, as many apertures as possible, separated from each other bysufficient space, are provided so that such a virtual opaque wall iscreated. For any camera array assembly, the number and placement ofopaque layers needed to form such a virtual vertical opaque wall can bedetermined through a ray tracing analysis.

In a further embodiment shown schematically in FIG. 2E, opticalcrosstalk suppression is achieved using spacers 295 constructed fromopaque materials. In another further embodiment shown schematically inFIG. 2F, optical crosstalk suppression is achieved using spacers 296coated with an opaque coating 297. The embodiments illustrated in FIGS.2E and 2F include stacked apertures 294 similar to the stacked apertures294 illustrated in FIG. 2D. In several embodiments, optical crosstalksuppression is achieved without using stacked apertures. In manyembodiments, any of a variety of light blocking materials can be used inthe construction or coating of spacers to achieve optical isolation.

Lens Properties

FIGS. 3A and 3B are diagrams illustrating changes in the height t of alens element pursuant to changes in dimensions in an x-y plane. A lenselement 320 in FIG. 3B is scaled by 1/n compared to a lens element 310in FIG. 3A. Note that during scaling it is important to keep the same F#so image properties don't change. As the diameter L/n of the lenselement 320 is smaller than the diameter L by a factor of n, the heighttin of the lens element 320 is also smaller than the height t of thelens element 310 by a factor of n. Hence, by using an array of smallerlens elements, the height of the camera array assembly can be reducedsignificantly. The reduced height of the camera array assembly may beused to design less aggressive lenses having better optical propertiessuch as improved chief ray angle, reduced distortion, and improved coloraberration.

FIG. 3C illustrates improving a chief ray angle (CRA) by reducing thethickness of the camera array assembly. CRA1 is the chief ray angle fora single lens covering an entire camera array. Although the chief rayangle can be reduced by increasing the distance between the camera arrayand the lens, the thickness constraints impose constraints on increasingthe distance. Hence, the CRA1 for camera array having a single lenselement is large, resulting in reduced optical performance. CRA2 is thechief ray angle for an imager in the camera array that is scaled inthickness as well as other dimensions. The CRA2 remains the same as theCRA1 of the conventional camera array and results in no improvement inthe chief ray angle. By modifying the distance between the imager andthe lens element as illustrated in FIG. 3C, however, the chief ray angleCRA3 in the camera array assembly may be reduced compared to CRA1 orCRA2, resulting in better optical performance. As described above, thecamera arrays according to the present invention has reduced thicknessrequirements, and therefore, the distance of the lens element and thecamera array may be increased to improve the chief ray angle. Thisrelaxed CRA, in turn, results in a lower F# and improved ModulationTransfer Function (MTF).

Specifically, one of the issues raised in camera design is how tocorrect for field curvature. An image projected through a lens is notplanar, but has an inherently curved surface. One way to correct thisfield curvature is to position a thick negative lens element 312 closeto or directly on the imager surface 314. The negative lens elementplanarizes the various angled beams of light 316 from the image, therebyaddressing the field curvature problem. Such field flattened imagesprovide superior image performance, allow for the manufacture of arraycameras with relaxed TTL requirements, and deliver very homogeneous MTF.However, one problem with this approach is that this field flatteningapproach intrinsically requires a high CRA. This makes the techniqueunsuitable for most cameras; however, camera arrays in accordance withembodiments of the invention allow for the use of backside imaging(BSI). Positioning the image sensor behind the substrate relaxes the CRAangle requirement, thereby allowing for the use of the negative lenselement field flattening approach shown in FIG. 3D.

Another advantage of the array camera relates to chromatic aberrations.Specifically, in a conventional polychromatic lens, the lens has to becorrected for chromatic aberrations, because the focal length throughthe lens is different for different wavelengths of light. As a result,it is necessary to compromise the performance of the lens for some ofthe color wavelengths to get acceptable overall color performance. Bymaking each optical channel narrow spectral band, color aberration isreduced and/or prevented, and each lens may be optimized to a specificcolor wavelength. For example, an imager receiving visible or near-IRspectrum may have a lens element specifically optimized for thisspectral band of light. For imagers detecting other light spectrum, thelens element may be constructed with different properties, such as radiiof curvature, so that a constant focal length across all wavelengths oflight is achieved so that, in turn, the focal plane is the same fordifferent spectral bands of light. The matching of the focal planeacross different wavelengths of light increases the sharpness of imagecaptured at the imager and reduces longitudinal chromatic aberration.Because each lens element may be designed to direct a narrow band oflight, the concomitant lack of color aberration means that the lenselements can be subject to less rigorous design constraints, yet producebetter or equivalent performance compared to a conventional lens elementcovering a wide light spectrum. In particular, there is no need toundertake costly aberration balancing correction. What is more, simplelenses generally have better MTF and lower F# (higher sensitivity). Itshould be noted that although the lenses used in these array camerashave much smaller color aberrations when compared to conventionalpolychromatic lenses, each lens is still designed to focus a certainwavelength-bandwidth. Accordingly, in one embodiment each of these“monochromatic” lenses can be optimally color corrected by usingcombinations of high and low Abbe number materials (different opticaldispersions).

Light of different wavelengths having different focal lengths(longitudinal color aberration) is not the only type of aberration thatoccurs in polychromatic optical systems. The refractive index of a lensis dependent on the wavelength of light passing through the lens. As aresult, a lens will impart different magnification to colors ofdifferent wavelengths. For example, the red wavelength band might have aslightly smaller magnification than green, and green may in turn have aslightly smaller magnification than blue. If the images obtained fromthese different wavelengths of light are then overlaid withoutcorrection, the image will lose resolution because the different colorswill not overlap correctly. Based on the properties of the material, thedifferential lateral distortions of the color magnification can bedetermined and then corrected. Correction can be accomplished byrestricting the profiles of the lenses so that each color has the samemagnification, but this reduces the possible degrees of freedomavailable for lens manufacture, and reduces the ability to optimize MTF.Accordingly, lateral distortion can be permitted optically, and thencorrected after imaging computationally. The electronic correction ofthe lateral color of the lens can actually provide improvements tosystem performance above and beyond simply correcting for the originaldistortion, because such correction directly improves the resolution ofthe system in terms of polychromatic MTF. In particular, lateral coloraberrations in a lens can be seen as a color dependent distortion of thelens. By mapping all differently distorted single color images of anobject back to the same rectangle, perfect overlap can be achieved inthe full color image resulting in the polychromatic MTF being the sameas the monochromatic one (not only due to the individual color channelcolor-blur correction, but also as a result of the exact superpositionof the different colors).

Yet another advantage to using many lenses, each optimized for use witha narrow band of light, is that the there is no restriction on the typeof lens that may be used. In particular, the array camera allows for theuse of diffractive, refractive, Fresnel lenses, or combinations of thesetypes of lenses. Diffractive lenses are attractive because they allowfor the creation of complex wavefronts with an essentially flat opticalelement, and they are also relatively simple to manufacture. Inconventional cameras it is not possible to use diffractive lensesbecause having a single imager means that the lens must be able toefficiently transmit a wide spectrum of light, and while diffractivelenses are very efficient at transmitting narrow wavelength bands oflight, there is a steep drop-off in performance for wavelengths of lightoutside of this optimized range. Because each array of the currentcamera may be focused on a narrow wavelength of light, the narrowoptimized wavelength band of these diffractive lenses is not a limitingfactor.

Other advantages of smaller lens elements include, among others, reducedcost, reduced amount of materials, and the reduction in themanufacturing steps. By providing n² lenses that are 1/n the size in xand y dimension (and thus 1/n thickness), the wafer size for producingthe lens element may also be reduced. This reduces the cost and theamount of materials considerably. Further, the number of lens substratesis reduced, which results in a reduced number of manufacturing steps andreduced attendant yield costs. The placement accuracy required toregister the lens array to the imagers is typically no more stringentthan in the case of a conventional imager because the pixel size for thecamera array according to the present invention may be substantially thesame as a conventional image sensor. In addition, monochromaticaberrations scale with lens diameter. Because array cameras are able touse smaller lenses, any aberrations that exist are smaller so it ispossible to use lenses with simpler profiles. This results in a systemthat is simultaneously better and less costly to fabricate. Smallersized lenses also have a lower volume, which results in lower sag orshrinkage during manufacture. Shrinkage is bad for replication becauseit deforms the desired lens profile, and results in the need for thefabricator to precompensate for the predicted level of sag so that thefinal lens shape will be correct. This precompensation is difficult tocontrol. With lower sag/shrinkage it is not necessary to have thesetight fabrication controls, again lowering the overall cost of themanufacture of the lenses.

In one embodiment, the WLO fabrication process includes: (i)incorporating lens element stops by plating the lens element stops ontothe substrate before lens molding, and (ii) etching holes in thesubstrate and performing two-sided molding of lenses through thesubstrate. The etching of holes in the substrate is advantageous becauseindex mismatch is not caused between plastic and substrate. In this way,light absorbing substrate that forms natural stops for all lens elements(similar to painting lens edges black) may be used.

In one embodiment, filters are part of the imager. In anotherembodiment, filters are part of a WLO subsystem. In an embodimentincluding a filter, it is preferred to dispose the filter (whether CFA,IR and/or VIS) into or close to the aperture stop surface and not at theimager sensor surface, because when positioned at a distance from theimager sensor small defects in those filter layers are averaged out overall entrance pupil positions, and are therefore less visible.

Imaging System and Processing Pipeline

FIG. 4 is a functional block diagram illustrating an imaging system 400,according to one embodiment. The imaging system 400 may include, amongother components, the camera array 410, an image processing pipelinemodule 420 and a controller 440. The camera array 410 includes two ormore imagers, as described above in detail with reference to FIGS. 1 and2. Images 412 are captured by the two or more imagers in the cameraarray 410.

The controller 440 is hardware, software, firmware or a combinationthereof for controlling various operation parameters of the camera array410. The controller 440 receives inputs 446 from a user or otherexternal components and sends operation signals 442 to control thecamera array 410. The controller 440 may also send information 444 tothe image processing pipeline module 420 to assist processing of theimages 412.

The image processing pipeline module 420 is hardware, firmware, softwareor a combination for processing the images received from the cameraarray 410. The image processing pipeline module 420 processes multipleimages 412, for example, as described below in detail with reference toFIG. 5. The processed image 422 is then sent for display, storage,transmittal or further processing.

FIG. 5 is a functional block diagram illustrating the image processingpipeline module 420, according to one embodiment. The image processingpipeline module 420 may include, among other components, an upstreampipeline processing module 510, an image pixel correlation module 514, aparallax confirmation and measurement module 518, a parallaxcompensation module 522, a super-resolution module 526, an addressconversion module 530, an address and phase offset calibration module554, and a downstream color processing module 564.

The address and phase offset calibration module 554 is a storage devicefor storing calibration data produced during camera arraycharacterization in the manufacturing process or a subsequentrecalibration process. In several embodiments, the calibration data canindicate mapping between the addresses of physical pixels 572 in theimagers and the logical addresses 546, 548 of an image. In otherembodiments, a variety of calibration data appropriate to a specificapplication can be utilized in the address and phase offset calibrationmodule.

The address conversion module 530 performs normalization based on thecalibration data stored in the address and phase offset calibrationmodule 554. Specifically, the address conversion module 530 converts“physical” addresses of the individual pixels in the image to “logical”addresses 548 of the individual pixels in the imagers or vice versa. Inorder for super-resolution processing to produce an image of enhancedresolution, the phase difference between corresponding pixels in theindividual imagers needs to be resolved. The super-resolution processmay assume that for each pixel in the resulting image the set of inputpixels from each of the imager is consistently mapped and that the phaseoffset of the image captured by each imager is already known withrespect to the position of the pixel in the resulting image.Alternatively, the pixel offsets can be estimated prior to thesuperresolution process. The address conversion module 530 resolves suchphase differences by converting the physical addresses in the images 412into logical addresses 548 of the resulting image for subsequentprocessing.

The images 412 captured by the imagers 540 are provided to the upstreampipeline processing module 510. The upstream pipe processing module 510may perform one or more of normalization of the color planes, BlackLevel calculation and adjustments, fixed noise compensation, optical PSF(point spread function) deconvolution, noise reduction, lateral colorcorrection and crosstalk reduction.

In one embodiment, the upstream pipeline processing module also performstemperature normalization. Temperature normalization corrects forchanges in the refractive index of the optical components through whichthe imagers receive light that result from changes in the temperature ofthe camera during use. In several embodiments, the temperaturenormalization process involves determining the temperature of the cameraarray by measuring the dark current of one or an average of a number ofthe camera array's imagers. Using this measurement, a refractive indexnormalization is performed by picking the correct point spread functionfrom temperature calibration data. Different point spread functions maybe obtained during a temperature dependent refractive indexcharacterization of the camera during manufacture, and stored in theimaging system for use in the temperature normalization process.

After the image is processed by the upstream pipeline processing module510, an image pixel correlation module 514 performs calculation toaccount for parallax that becomes more apparent as objects beingcaptured approach the camera array. Specifically, the image pixelcorrelation module 514 aligns portions of images captured by differentimagers to compensate for the parallax. In one embodiment, the imagepixel correlation module 514 compares the difference between the averagevalues of neighboring pixels with a threshold and flags the potentialpresence of parallax when the difference exceeds the threshold. Thethreshold may change dynamically as a function of the operatingconditions of the camera array. Further, the neighborhood calculationsmay also be adaptive and reflect the particular operating conditions ofthe selected imagers.

The image is then processed by the parallax confirmation and measurementmodule 518 to detect and meter the parallax. In one embodiment, parallaxdetection is accomplished by a running pixel correlation monitor. Thisoperation takes place in logical pixel space across the imagers withsimilar integration time conditions. When the scene is at practicalinfinity, the data from the imagers is highly correlated and subjectonly to noise-based variations. When an object is close enough to thecamera, however, a parallax effect is introduced that changes thecorrelation between the imagers. Due to the spatial layout of theimagers, the nature of the parallax-induced change is consistent acrossall imagers. Within the limits of the measurement accuracy, thecorrelation difference between any pair of imagers dictates thedifference between any other pair of imagers and the differences acrossthe other imagers. This redundancy of information enables highlyaccurate parallax confirmation and measurement by performing the same orsimilar calculations on other pairs of imagers. If parallax is presentin the other pairs, the parallax should occur at roughly the samephysical location of the scene taking into account the positions of theimagers. The measurement of the parallax may be accomplished at the sametime by keeping track of the various pair-wise measurements andcalculating an “actual” parallax difference as a least squares (orsimilar statistic) fit to the sample data. Other methods for detectingthe parallax may include detecting and tracking vertical and horizontalhigh-frequency image elements from frame-to-frame.

The parallax compensation module 522 processes images including objectsclose enough to the camera array to induce parallax differences largerthan the accuracy of the phase offset information required by superresolution process. The parallax compensation module 522 uses thescan-line based parallax information generated in the parallax detectionand measurement module 518 to further adjust mapping between physicalpixel addresses and logical pixel addresses before the super-resolutionprocess. There are two cases that occur during this processing. In amore common case, addressing and offsetting adjustment are required whenthe input pixels have shifted positions relative to theimage-wise-corresponding pixels in other imagers. In this case, nofurther processing with respect to parallax is required beforeperforming super-resolution. In a less common case, a pixel or group ofpixels are shifted in such a way that exposes the occlusion set. In thiscase, the parallax compensation process generates tagged pixel dataindicating that the pixels of the occlusion set should not be consideredin the super-resolution process.

After the parallax change has been accurately determined for aparticular imager, the parallax information 524 is sent to the addressconversion module 530. The address conversion module 530 uses theparallax information 524 along with the calibration data 558 from theaddress and phase offset calibration module 554 to determine theappropriate X and Y offsets to be applied to logical pixel addresscalculations. The address conversion module 530 also determines theassociated sub-pixel offset for a particular imager pixel with respectto pixels in the resulting image 428 produced by the super-resolutionprocess. The address conversion module 530 takes into account theparallax information 524 and provides logical addresses 546 accountingfor the parallax.

After performing the parallax compensation, the image is processed bythe super-resolution module 526 to obtain a high resolution synthesizedimage 422 from low resolution images, as described below in detail. Thesynthesized image 422 may then be fed to the downstream color processingmodule 564 to perform one or more of the following operations: focusrecover, white balance, color correction, gamma correction, RGB to YUVcorrection, edge-aware sharpening, contrast enhancement and compression.

The image processing pipeline module 420 may include components foradditional processing of the image. For example, the image processingpipeline module 420 may include a correction module for correctingabnormalities in images caused by a single pixel defect or a cluster ofpixel defects. The correction module may be embodied on the same chip asthe camera array, as a component separate from the camera array or as apart of the super-resolution module 526.

Super-Resolution Processing

In one embodiment, the super-resolution module 526 generates a higherresolution synthesized image by processing low resolution imagescaptured by the imagers 540. The overall image quality of thesynthesized image is higher than images captured from any one of theimagers individually. In other words, the individual imagers operatesynergistically, each contributing to higher quality images using theirability to capture a narrow part of the spectrum without sub-sampling.The image formation associated with the super-resolution techniques maybe expressed as follows:

y _(k) =W _(k) ·x+n _(k) , ∀k=1 . . . p  equation (2)

where W_(k) represents the contribution of the HR scene (x) (viablurring, motion, and sub-sampling) to each of the LR images (y_(k))captured on each of the k imagers and n_(k) is the noise contribution.

Imager Configurations

FIGS. 6A through 6F illustrate various configurations of imagers forobtaining a high resolution image through a super-resolution process,according to embodiments of the present invention. In FIGS. 6A through6F, “R” represents an imager having a red filter, “G” represents aimager having a green filter, “B” represents an imager having a bluefilter, “P” represents a polychromatic imager having sensitivity acrossthe entire visible spectra and near-IR spectrum, and “I” represents animager having a near-IR filter. The polychromatic imager may sampleimage from all parts of the visible spectra and the near-IR region(i.e., from 650 nm to 800 nm). In the embodiment of FIG. 6A, the centercolumns and rows of the imagers include polychromatic imagers. Theremaining areas of the camera array are filled with imagers having greenfilters, blue filters, and red filters. The embodiment of FIG. 6A doesnot include any imagers for detecting near-IR spectrum alone.

The embodiment of FIG. 6B has a configuration similar to conventionalBayer filter mapping. This embodiment does not include any polychromaticimagers or near-IR imagers. As described above in detail with referenceto FIG. 1, the embodiment of FIG. 6B is different from conventionalBayer filter configuration in that each color filter is mapped to eachimager instead of being mapped to an individual pixel.

FIG. 6C illustrates an embodiment where the polychromatic imagers form asymmetric checkerboard pattern. FIG. 6D illustrates an embodiment wherefour near-IR imagers are provided. FIG. 6E illustrates an embodimentwith irregular mapping of imagers. FIG. 6F illustrates an embodimentwhere a 5×5 sensor array is organized into 17 imagers having greenfilters, four imagers having red filters, and four imagers having bluefilters. The sensors are distributed symmetrically around the centralaxis of the imaging array. As is discussed further below, distributingthe imagers in this way prevents pixels that can be imaged by a sensorfrom being occluded from sensors capturing other wavelengths of light.The embodiments of FIGS. 6A through 6F are merely illustrative andvarious other layouts of imagers can also be used.

The use of polychromatic imagers and near-IR imagers is advantageousbecause these sensors may capture high quality images in low lightingconditions. The images captured by the polychromatic imager or thenear-IR imager are used to denoise the images obtained from regularcolor imagers. However, as discussed above, these polychromatic lensesrequire that an associated color correction technique be used to addresscolor aberrations inherent in a single lens trying to capture allwavelengths of light and deliver it to the same focal plane. Anyconventional color correction technique may be utilized with theproposed array cameras.

Imager Layout

The premise of increasing resolution by aggregating multiple lowresolution images relies upon the different low resolution imagesrepresenting slightly different viewpoints of the same scene. If the LRimages are all shifted by integer units of a pixel, then each imagecontains essentially the same information. Therefore, there is no newinformation in the LR images that can be used to create a HR image. Incamera arrays according to embodiments of the invention, the layout ofthe imagers in the array may be preset and controlled so that eachimager in a row or a column captures an image that is shifted a fixedsub-pixel distance relative to the images captured by its neighboringimagers. Ideally, the images captured by each imager are spatiallyoffset from the other imagers in such a way as to provide uniformsampling of the scene or the light field and the uniformity of samplingis such that the LR images captured by each of the imagers yieldsnon-redundant information about the sampled scene (light field). Suchnon-redundant information about the scene can be utilized by subsequentsignal processing processes to synthesize a single HR image.

A sub-pixel shift between the images captured by two imagers is not,however, sufficient to ensure uniformity of sampling. The uniformity ofsampling or sampling diversity of two imagers is a function of objectdistance. The sampled space by pixels of a pair of imagers isillustrated in FIG. 6G. A first set of rays (610) map to pixels ofimager A, while a second set of rays (620) map to pixels of imager B.Conceptually, two adjacent rays from a given imager define the part ofthe object space that is sampled by a specific pixel in that imager. Atdistance z1 from the camera plane, there is sufficient samplingdiversity since the rays of the pixels of imager A are partially offsetfrom the rays of the pixels of imager B. As the distance decreases thereare specific distances (z2, z3, z4) where there is no sampling diversitybetween imager A and imager B. The lack of sampling diversity betweenthe two imagers quite simply implies that there is no additionalinformation in the scene captured by imager B as compared to thatcaptured by imager A. As is discussed further below, an increased numberof imagers in an array camera can mitigate the impact of objectdistances at which pair of imagers' sample space fully overlap. When apair of imagers lack sampling diversity, the other imagers in the arrayprovide the necessary sampling diversity to achieve resolutionenhancement. Consequently, the ability of an imager system utilizing a2×2 array of imagers to achieve superresolution is typically morelimited than camera systems in accordance with embodiments of theinvention that use a larger camera array.

Referring back to the camera array structures illustrated in FIGS. 2A-2Dthe wafer level optics includes a plurality of lens elements, where eachlens element covers one of the sensors in the array. The physical layoutof pixels in a single imager of a camera array in accordance with anembodiment of the invention is illustrated in FIG. 6H. The imager is anarray of pixels 650 overlaid with color filters 652 and microlenses 654.The microlenses that sit on top of the color filters are used to focuslight on the active area of each underlying pixel. The microlenses canbe thought of as sampling the continuous light field in object spacesampled by the main lens. Whereas the main lens samples the sceneradiance light field, the micro-lenses sample the sensor irradiancelight field.

The main lens associated with each imager maps the points in the objectspace to points in the image space such at that the mapping is bijective(onto-to-one and onto). Each microlens samples a finite extent of thesensor irradiance light field. The sensor irradiance light field iscontinuous and is the result of a bijective mapping from the objectspace. Thus, the microlens sampling of a finite extent of the sensorirradiance light field is also a sampling of a corresponding finiteextent of the scene radiance light field in object space.

Moving the microlens by a small amount δ laterally along the plain ofthe imager pixels changes the sampled object space at a certain distancez_(k) by a correspondingly appropriate factor δ. With an n×n (n>2) arraycamera, we can choose a baseline microlens shift can be determined bythe main lens profile (for example, the chief ray angle) for a baselineimager. For each of the other imagers that sample the same wavelength asthe baseline imager, the microlenses of each of the pixels in the imagerare shifted by a sub-pixel amount to sample a different part of thescene radiance light field. Thus for a set of imagers arranged in an n×ngrid, the sub-pixel shift for a imager that images the same wavelengthsas the baseline imager (1,1) at a grid location (i,j) (1≦i,j≦n) isgoverned by (δ_(x), δ_(y)) where,

${\frac{\left( { - 1} \right)}{n} \times {pixelsize}} < \delta_{x} \leq {\frac{}{n} \times {pixelsize}}$${\frac{\left( {j - 1} \right)}{n} \times {pixelsize}} < \delta_{y} \leq {\frac{j}{n} \times {pixelsize}}$

Many camera arrays in accordance with embodiments of the inventioninclude significantly more Green imagers than Red and Blue imagers. Forexample, the array camera illustrated in FIG. 6F includes 17 Greenimagers, 4 Red imagers, and 4 Blue imagers. For the purpose ofcalculating the sub-pixel shifts, the Green imagers can be treated as ann×n grid. Whereas the Red imagers and the Blue imagers can each betreated as a 2×2 grid for the purpose of calculating the sub-pixelshifts.

The sub-pixel shifts discussed above are determined relative to abaseline imager located at the corner of the grid, many embodiments ofthe invention utilize radial sub-pixel shifts from a baseline imagerlocated at the center of the sensor array. In several embodiments, theradial sub-pixel shifts are chosen so the sub-pixel shifts are evenlydistributed to enable the greatest sampling diversity.

The constraints on microlens sub-pixel shifts defined above achieve thehighest increases in diversity and can enable the greatest increases inresolution through superresolution processing. Sub-pixel shifts that donot satisfy the constraints, but still provide an increase in samplingdiversity can also be used to enable some increase in resolution throughsuperresolution processing. Therefore, embodiments of the invention arenot limited to microlens shifts that result in the greatest increases indiversity and in many instances utilize a variety of different microlensshift configurations that provide at least some increase in samplingdiversity and that are satisfactory for the requirements of a specificapplication.

Symmetry of Imager Placement in Camera Array

An issue of separating the spectral sensing elements into differentimagers is parallax caused by the physical separation of the imagers. Byensuring that the imagers are symmetrically placed, at least two imagerscan capture the pixels around the edge of a foreground object. In thisway, the pixels around the edge of a foreground object may be aggregatedto increase resolution as well as avoiding any occlusions. In theabsence of a symmetrical distribution, a pixel around the edge of aforeground object that is visible to a first imager, for example a Redimager, may be occluded to a second imager that captures differentwavelengths, for example a blue imager. Accordingly, color informationfor the pixel cannot be accurately reconstructed. By symmetricallydistributing the sensors, the likelihood that a foreground object willocclude pixels is significantly reduced.

Pixel occlusion caused by an asymmetric distribution of Red and Blueimagers in a simple array is illustrated in FIG. 6I. A pair of Redimagers 672 is located on the left hand side of the camera array 670 anda pair of Blue imagers 674 is located on the right hand side of thecamera array. A foreground object 676 is present and the Red imagers 672are capable of imaging regions beyond the foreground object on the lefthand side of the foreground object. However, the foreground objectoccludes the Red imagers from imaging these regions. Therefore, thearray camera is incapable of reconstructing color information for theseregions.

An array that includes a symmetric distribution of Red and Blue imagersin accordance with an embodiment of the invention is illustrated in FIG.6J. The camera array 780 includes a pair of Red imagers 782symmetrically distributed around the central axis of the camera arrayand a pair of Blue imagers 784 symmetrically distributed around thecentral axis of the camera array. Due to the even distribution, a Redimager and a Blue imager are both able to image beyond the foregroundobject 786 on the left hand side of the foreground object and a Redimager and a Blue imager are both able to image beyond the foregroundobject on the right hand side of the foreground object.

The symmetrical arrangement of the simple embodiment illustrated in FIG.6J can be generalized to array cameras including Red, Green, Blueimagers and/or additional polychromatic or near-IR cameras. Bydistributing each of the different types of imagers symmetrically aroundthe central axis of the camera array, the effects of parallax introducedby foreground objects can be significantly reduced and color artifactsthat would otherwise be introduced, avoided.

The effects of parallax on the sampling of color can also be reduced byusing parallax information in polychromatic imagers to improve theaccuracy of the sampling of color from the color filtered imagers.

Use of Near-IR Imagers to Obtain Improved High Resolution Images

In one embodiment, near-IR imagers are used to determine relativeluminance differences compared to a visible spectra imager. Objects havediffering material reflectivity results in differences in the imagescaptured by the visible spectra and the near-IR spectra. At low lightingconditions, the near-IR imager exhibits a higher signal to noise ratios.Therefore, the signals from the near-IR sensor may be used to enhancethe luminance image. The transferring of details from the near-IR imageto the luminance image may be performed before aggregating spectralimages from different imagers through the super-resolution process. Inthis way, edge information about the scene may be improved to constructedge-preserving images that can be used effectively in thesuper-resolution process. The advantage of using near-IR imagers isapparent from equation (2) where any improvement in the estimate for thenoise (i.e., n) leads to a better estimate of the original HR scene (x).

Generation of High Resolution Images

FIG. 7 is a flowchart illustrating a process of generating an HR imagefrom LR images captured by a plurality of imagers, according to oneembodiment. First, luma images, near-IR images and chroma images arecaptured 710 by imagers in the camera array. Then normalization isperformed 714 on the captured images. The images can be normalized in avariety of ways including but not limited to normalizing the colorplanes of the images, performing temperature compensation, and mappingphysical addresses of the imagers to logical addresses in the enhancedimage. In other embodiments, a variety of normalization processappropriate to the specific imagers and imaging applications. Parallaxcompensation is then performed 720 to resolve any differences in thefield-of-views of the imagers due to spatial separations between theimagers. Super-resolution processing is then performed 724 to obtainsuper-resolved luma images, super-resolved near-IR images, andsuper-resolved chroma images.

Then it is determined 728 if the lighting condition is better than apreset parameter. If the lighting condition is better than theparameter, the process proceeds to normalize 730 a super-resolvednear-IR image with respect to a super-resolved luma image. A focusrecovery is then performed 742. In one embodiment, the focus recovery isperformed 742 using PSF (point spread function) deblurring per eachcolor channel. Then the super-resolution is processed 746 based onnear-IR images and the luma images. A synthesized image is thenconstructed 750.

If it is determined 728 that the lighting condition is not better thanthe preset parameter, the super-resolved near-IR images and luma imagesare aligned 734. Then the super-resolved luma images are denoised 738using the near-IR super-resolved images. Then the process proceeds toperforming focus recovery 742 and repeats the same process as when thelighting condition is better than the preset parameter. Then the processterminates.

Normalization of Color Planes

The relative response of each of the Red, Green, Blue imagers across theimaging planes varies. The variance can be the result of many factorsincluding the optical alignment of the lens and asymmetrical sensorlight path geometry. For a given lens and imager, the variance can becompensated for by calibration and normalization. Without compensation,the variance can give rise to artifacts such as color shading.

A process for normalizing a imager with respect to a baseline imager,which is typically a Green imager located in the center of the cameraarray, in accordance with an embodiment of the invention is discussedbelow with reference to the normalization of a Red imager with respectto a baseline Green imager. A similar process can be used to normalizeBlue imagers with respect to a baseline Green imager. In manyembodiments, the process is applied to normalize each Red and Blueimager in a camera array.

A normalization surface can be calibrated by first capturing a scenewith flat reflectance, and calculating a color ratio surface to serve asthe basis for normalization. An ideal normalization surface is uniformand can be described as:

Color Ratio G/R=G(i,j)/R(i,j)=K=G _(center) /R _(center)

where (i,j) describe the pixel position, K is a constant, andG_(center), and R_(center), describe the pixel value at the centerposition.

The output pixel values of the calibration scene contain the ideal pixelvalues plus noise plus black level offset, and can be described asfollows:

SR(i,j)=R(i,j)+Noise R(i,j)+black offset

SG(i,j)=G(i,j)+Noise G(i,j)+black offset

where SR, and SG are the output pixel values from each imager.

A process for calibrating the sensor in accordance with an embodiment ofthe invention is illustrated in FIG. 7A. The process 760 includesremoving (762) the black level offset from the sensor pixel values, andlow pass filtering (764) the image planes to reduce noise. Thenormalization plane is calculated (766) and several embodiments arecalculated as follows:

Norm R=G(i,j)/(R(i,j)×(G _(center) /R _(center)))

where G_(center), and R_(center), are the pixel values at the centerposition.

Following the calculation of the normalization plane, an averagingfilter can be applied (768) and the values of the Norm R plane arestored (770).

The cost of carrying all of the normalization data for each of thesensors in a sensor array can be quite high. Therefore, many embodimentsscan the Norm R plane using a space filling curve to form a onedimensional array. The resultant one dimensional array can be modeled ina variety of different ways including being modeled as a polynomial withsuitable order. In several embodiments, the polynomials of the fittedpolynomial are stored (810) as parameters that are used duringcalibration to reconstruct the two dimensional normalization plane. Theconstruction of a space filling curve in accordance with severalembodiments of the invention is discussed further below.

In several embodiments, a space filling curve is used to form a onedimensional array describing a normalization plane. A space fillingcurve, which is constructed using a spiral scan, is illustrated in FIG.7B. A space filling curve 780 can be constructed by starting at thecenter of the normalization plane 781 and traversing a four sided squareoutwards. Each side of the square expands by two pixels compared to theprevious square such that every pixel will be traversed exactly once. Inthe illustrated embodiment, each position 782 which is marked with an‘X’ corresponds to a valid pixel position. The imager may not have asquare geometry, so the scan path may traverse empty space (indicated asdashed lines). For each position traversed, if it is a valid pixelposition, a new data entry is added to the one dimensional data array.Otherwise, the traversing continues without adding a new value to thedata array. In many embodiments, the one dimensional data array can beefficiently approximated using a 6^(th) order polynomial that can berepresented using the seven coefficients of the polynomial. Given thatcalibration data is typically required for each Red and Blue imager,expressing the normalization planes as coefficients of a polynomialrepresents a significant reduction in storage requirements. In manyembodiments, higher or lower order polynomials, other functions, and/orother compressed representations are utilized to represent thenormalization plane in accordance with the requirements of a specificapplication.

The data value along each side exhibits a fixed geometric relationship.The optical path to the focal point of the lens is shorter for the cellsnear the center line. The base sensitivity can be thought of as a onedimensional center cut of the calibration surface and approximated by alow order polynomial. The sensitivity polynomial can be either stored asa machine constant (i.e., common to all devices with the same design),or stored along with the scan polynomial to provide additionalflexibility. Accordingly, many embodiments of the invention adjust thepixel value based upon the distance factor as follows. For each sidescan, one of the coordinates will be a constant, i.e., constant ‘y’ forhorizontal scan and constant ‘x’ for vertical scan. For each pixel inthe side scan, the sensitivity factor is adjusted towards the constant‘x’ or ‘y’ distance.

By way of example, for a horizontal scan the base value can be found byevaluating the sensitivity polynomial based on the distance ‘y’ from thecenter. In many embodiments, a suitable polynomial is a fourth orderpolynomial. Although other polynomials and/or other functions can beutilized in accordance with the requirements of a specific application.For each pixel in the scan path, the distance from the surface origin isused to find the corresponding sensitivity from the polynomial in thesame manner. The pixel value is multiplied by an adjustment factor andthen stored in the scanned data array. This adjustment factor iscalculated by dividing the base value with the current sensitivityvalue. For the vertical scan a similar method can be applied. Althoughthe example uses a polynomial based sensitivity adjustment, othersensitivity functions and/or adjustments can be utilized depending uponthe requirements of a specific application in accordance with variousembodiments of the invention.

Once calibration data has been obtained for an imager, the calibrationdata can be used in the normalization of pixel information captured bythe imager. The process typically involves retrieving the storedcalibration data, removing the black offset from the captured image andmultiplying the resultant values with the normalization plane. When thenormalization plane is expressed as a polynomial in the manner outlinedabove, the polynomial is used to generate a one-dimensional array and aninverse scan of the one-dimensional array is used to form the twodimensional normalization plane. Where a sensitivity adjustment wasapplied during calibration, an adjustment factor is calculated that isthe reciprocal of the adjustment factor applied during the calibrationscan and the adjustment factor is applied to the values in the onedimensional array during the inverse scan. When other space fillingcurves, representations of the resulting one dimensional data array,and/or sensitivity adjustments are performed during the calibrationprocess, the normalization process is adjusted accordingly.

As can be readily appreciated, calibration and normalization processesin accordance with embodiments of the invention can be applied to eachof the Red and Blue imagers in the camera array. In many embodiments, aGreen imager located in the center of the camera array is used whenperforming the calibration. In other embodiments, a different Greenimager and/or multiple Green imagers can be utilized in the calibrationof the Red and Blue imagers in the camera array.

Image Fusion of Color Images with Near-IR Images

The spectral response of CMOS imagers is typically very good in thenear-IR regions covering 650 nm to 800 nm and reasonably good between800 nm and 1000 nm. Although near-IR images having no chromainformation, information in this spectral region is useful in lowlighting conditions because the near-IR images are relatively free ofnoise. Hence, the near-IR images may be used to denoise color imagesunder the low lighting conditions.

In one embodiment, an image from a near-IR imager is fused with anotherimage from a visible light imager. Before proceeding with the fusion, aregistration is performed between the near-IR image and the visiblelight image to resolve differences in viewpoints. The registrationprocess may be performed in an offline, one-time, processing step. Afterthe registration is performed, the luminance information on the near-IRimage is interpolated to grid points that correspond to each grid pointon the visible light image.

After the pixel correspondence between the near-IR image and the visiblelight image is established, a denoising and detail transfer process maybe performed. The denoising process allows transfer of signalinformation from the near-IR image to the visible light image to improvethe overall SNR of the fusion image. The detail transfer ensures thatedges in the near-IR image and the visible light image are preserved andaccentuated to improve the overall visibility of objects in the fusedimage.

In one embodiment, a near-IR flash may serve as a near-IR light sourceduring capturing of an image by the near-IR imagers. Using the near-IRflash is advantageous, among other reasons, because (i) the harshlighting on objects of interest may be prevented, (ii) ambient color ofthe object may be preserved, and (iii) red-eye effect may be prevented.

In one embodiment, a visible light filter that allows only near-IR raysto pass through is used to further optimize the optics for near-IRimaging. The visible light filter improves the near-IR optics transferfunction because the light filter results in sharper details in thenear-IR image. The details may then be transferred to the visible lightimages using a dual bilateral filter as described, for example, in EricP. Bennett et al., “Multispectral Video Fusion,” Computer Graphics (ACMSIGGRAPH Proceedings) (Jul. 25, 2006), which is incorporated byreference herein in its entirety.

Dynamic Range Determination by Differing Exposures at Imagers

An auto-exposure (AE) algorithm is important to obtaining an appropriateexposure for the scene to be captured. The design of the AE algorithmaffects the dynamic range of captured images. The AE algorithmdetermines an exposure value that allows the acquired image to fall inthe linear region of the camera array's sensitivity range. The linearregion is preferred because a good signal-to-noise ratio is obtained inthis region. If the exposure is too low, the picture becomesunder-saturated while if the exposure is too high the picture becomesover-saturated. In conventional cameras, an iterative process is takento reduce the difference between measured picture brightness andpreviously defined brightness below a threshold. This iterative processrequires a large amount of time for convergence, and sometimes resultsin an unacceptable shutter delay.

In one embodiment, the picture brightness of images captured by aplurality of imagers is independently measured. Specifically, aplurality of imagers are set to capturing images with differentexposures to reduce the time for computing the adequate exposure. Forexample, in a camera array with 5×5 imagers where 8 luma imagers and 9near-IR imagers are provided, each of the imagers may be set withdifferent exposures. The near-IR imagers are used to capture low-lightaspects of the scene and the luma imagers are used to capture the highillumination aspects of the scene. This results in a total of 17possible exposures. If exposure for each imager is offset from anadjacent imager by a factor of 2, for example, a maximum dynamic rangeof 2¹⁷ or 102 dB can be captured. This maximum dynamic range isconsiderably higher than the typical 48 dB attainable in a conventionalcamera with 8 bit image outputs.

At each time instant, the responses (under-exposed, over-exposed oroptimal) from each of the multiple imagers are analyzed based on howmany exposures are needed at the subsequent time instant. The ability toquery multiple exposures simultaneously in the range of possibleexposures accelerates the search compared to the case where only oneexposure is tested at once. By reducing the processing time fordetermining the adequate exposure, shutter delays and shot-to-shot lagsmay be reduced.

In one embodiment, the HDR image is synthesized from multiple exposuresby combining the images after linearizing the imager response for eachexposure. The images from the imagers may be registered before combiningto account for the difference in the viewpoints of the imagers.

In one embodiment, at least one imager includes HDR pixels to generateHDR images. HDR pixels are specialized pixels that capture high dynamicrange scenes. Although HDR pixels show superior performances compared toother pixels, HDR pixels show poor performance at low lightingconditions in comparison with near-IR imagers. To improve performance atlow lighting conditions, signals from the near-IR imagers may be used inconjunction with the signal from the HDR imager to attain better qualityimages across different lighting conditions.

In one embodiment, an HDR image is obtained by processing imagescaptured by multiple imagers by processing, as disclosed, for example,in Paul Debevec et al., “Recovering High Dynamic Range Radiance Mapsfrom Photographs,” Computer Graphics (ACM SIGGRAPH Proceedings), (Aug.16, 1997), which is incorporated by reference herein in its entirety.The ability to capture multiple exposures simultaneously using theimager is advantageous because artifacts caused by motion of objects inthe scene can be mitigated or eliminated.

Hyperspectral Imaging by Multiple Imagers

In one embodiment, a multi-spectral image is rendered by multipleimagers to facilitate the segmentation or recognition of objects in ascene. Because the spectral reflectance coefficients vary smoothly inmost real world objects, the spectral reflectance coefficients may beestimated by capturing the scene in multiple spectral dimensions usingimagers with different color filters and analyzing the captured imagesusing Principal Components Analysis (PCA).

In one embodiment, half of the imagers in the camera array are devotedto sampling in the basic spectral dimensions (R, G, and B) and the otherhalf of the imagers are devoted to sampling in a shifted basic spectraldimensions (R′, G′, and B′). The shifted basic spectral dimensions areshifted from the basic spectral dimensions by a certain wavelength(e.g., 10 nm).

In one embodiment, pixel correspondence and non-linear interpolation isperformed to account for the sub-pixel shifted views of the scene. Thenthe spectral reflectance coefficients of the scene are synthesized usinga set of orthogonal spectral basis functions as disclosed, for example,in J. P. S. Parkkinen, J. Hallikainen and T. Jaaskelainen,“Characteristic Spectra of Munsell Colors,” J. Opt. Soc. Am., A 6:318(August 1989), which is incorporated by reference herein in itsentirety. The basis functions are eigenvectors derived by PCA of acorrelation matrix and the correlation matrix is derived from a databasestoring spectral reflectance coefficients measured by, for example,Munsell color chips (a total of 1257) representing the spectraldistribution of a wide range of real world materials to reconstruct thespectrum at each point in the scene.

At first glance, capturing different spectral images of the scenethrough different imagers in the camera array appears to traderesolution for higher dimensional spectral sampling. However, some ofthe lost resolution may be recovered. The multiple imagers sample thescene over different spectral dimensions where each sampling grid ofeach imager is offset by a sub-pixel shift from the others. In oneembodiment, no two sampling grid of the imager overlap. That is, thesuperposition of all the sampling grids from all the imagers forms adense, possibly non-uniform, montage of points. Scattered datainterpolation methods may be used to determine the spectral density ateach sample point in this non-uniform montage for each spectral image,as described, for example, in Shiaofen Fang et al., “Volume MorphingMethods for Landmark Based 3D Image Deformation” by SPIE vol. 2710,proc. 1996 SPIE Intl Symposium on Medical Imaging, page 404-415, NewportBeach, Calif. (February 1996), which is incorporated by reference hereinin its entirety. In this way, a certain amount of resolution lost in theprocess of sampling the scene using different spectral filters may berecovered.

As described above, image segmentation and object recognition arefacilitated by determining the spectral reflectance coefficients of theobject. The situation often arises in security applications wherein anetwork of cameras is used to track an object as it moves from theoperational zone of one camera to another. Each zone may have its ownunique lighting conditions (fluorescent, incandescent, D65, etc.) thatmay cause the object to have a different appearance in each imagecaptured by different cameras. If these cameras capture the images in ahyper-spectral mode, all images may be converted to the same illuminantto enhance object recognition performance.

In one embodiment, camera arrays with multiple imagers are used forproviding medical diagnostic images. Full spectral digitized images ofdiagnostic samples contribute to accurate diagnosis because doctors andmedical personnel can place higher confidence in the resultingdiagnosis. The imagers in the camera arrays may be provided with colorfilters to provide full spectral data. Such camera array may beinstalled on cell phones to capture and transmit diagnostic informationto remote locations as described, for example, in Andres W. Martinez etal., “Simple Telemedicine for Developing Regions: Camera Phones andPaper-Based Microfluidic Devices for Real-Time, Off-Site Diagnosis,”Analytical Chemistry (American Chemical Society) (Apr. 11, 2008), whichis incorporated by reference herein in its entirety. Further, the cameraarrays including multiple imagers may provide images with a large depthof field to enhance the reliability of image capture of wounds, rashes,and other symptoms.

In one embodiment, a small imager (including, for example, 20-500pixels) with a narrow spectral bandpass filters is used to produce asignature of the ambient and local light sources in a scene. By usingthe small imager, the exposure and white balance characteristics may bedetermined more accurately at a faster speed. The spectral bandpassfilters may be ordinary color filters or diffractive elements of abandpass width adequate to allow the number of camera arrays to coverthe visible spectrum of about 400 nm. These imagers may run at a muchhigher frame rate and obtain data (which may or may not be used for itspictorial content) for processing into information to control theexposure and white balance of other larger imagers in the same cameraarray. The small imagers may also be interspersed within the cameraarray.

Optical Zoom Implemented Using Multiple Imagers

In one embodiment, a subset of imagers in the camera array includestelephoto lenses. The subset of imagers may have other imagingcharacteristics that are the same as imagers with non-telephoto lenses.Images from this subset of imagers are combined and super-resolutionprocessed to form a super-resolution telephoto image. In anotherembodiment, the camera array includes two or more subsets of imagersequipped with lenses of more than two magnifications to providediffering zoom magnifications.

Embodiments of the camera arrays may achieve its final resolution byaggregating images through super-resolution. Taking an example ofproviding 5×5 imagers with a 3× optical zoom feature, if 17 imagers areused to sample the luma (G) and 8 imagers are used to sample the chroma(R and B), 17 luma imagers allow a resolution that is four times higherthan what is achieved by any single imager in the set of 17 imagers. Ifthe number of the imagers is increased from 5×5 to 6×6, an addition of11 extra imagers becomes available. In comparison with the 8 Megapixelconventional image sensor fitted with a 3× zoom lens, a resolution thatis 60% of the conventional image sensor is achieved when 8 of theadditional 11 imagers are dedicated to sampling luma (G) and theremaining 3 imagers are dedicated to chroma (R and B) and near-IRsampling at 3× zoom. This considerably reduces the chroma sampling (ornear-IR sampling) to luma sampling ratio. The reduced chroma to lumasampling ratio is somewhat offset by using the super-resolved luma imageat 3× zoom as a recognition prior on the chroma (and near-IR) image toresample the chroma image at a higher resolution.

With 6×6 imagers, a resolution equivalent to the resolution ofconventional image sensor is achieved at 1× zoom. At 3× zoom, aresolution equivalent to about 60% of conventional image sensoroutfitted with a 3× zoom lens is obtained by the same imagers. Also,there is a decrease in luma resolution at 3× zoom compared withconventional image sensors with resolution at 3× zoom. The decreasedluma resolution, however, is offset by the fact that the optics ofconventional image sensor has reduced efficiency at 3× zoom due tocrosstalk and optical aberrations.

The zoom operation achieved by multiple imagers has the followingadvantages. First, the quality of the achieved zoom is considerablyhigher than what is achieved in the conventional image sensor due to thefact that the lens elements may be tailored for each change in focallength. In conventional image sensors, optical aberrations and fieldcurvature must be corrected across the whole operating range of thelens, which is considerably harder in a zoom lens with moving elementsthan in a fixed lens element where only aberrations for a fixed focallength need to be corrected. Additionally, the fixed lens in the imagershas a fixed chief ray angle for a given height, which is not the casewith conventional image sensor with a moving zoom lens. Second, theimagers allow simulation of zoom lenses without significantly increasingthe optical track height. The reduced height allows implementation ofthin modules even for camera arrays with zooming capability.

The overhead required to support a certain level of optical zoom incamera arrays according to some embodiments is tabulated in Table 2.

TABLE 2 No. of Luma No. of Chroma No. of Imagers at Imagers at Imagersin different different Camera Zoom levels Zoom Levels array 1× 2× 3× 1×2× 3× 25 17 0 0 8 0 0 36 16 0 8 8 0 4

In one embodiment, the pixels in the images are mapped onto an outputimage with a size and resolution corresponding to the amount of zoomdesired in order to provide a smooth zoom capability from thewidest-angle view to the greatest-magnification view. Assuming that thehigher magnification lenses have the same center of view as the lowermagnification lenses, the image information available is such that acenter area of the image has a higher resolution available than theouter area. In the case of three or more distinct magnifications, nestedregions of different resolution may be provided with resolutionincreasing toward the center.

An image with the most telephoto effect has a resolution determined bythe super-resolution ability of the imagers equipped with the telephotolenses. An image with the widest field of view can be formatted in atleast one of two following ways. First, the wide field image may beformatted as an image with a uniform resolution where the resolution isdetermined by the super-resolution capability of the set of imagershaving the wider-angle lenses. Second, the wide field image is formattedas a higher resolution image where the resolution of the central part ofthe image is determined by the super-resolution capability of the set ofimagers equipped with telephoto lenses. In the lower resolution regions,information from the reduced number of pixels per image area isinterpolated smoothly across the larger number of “digital” pixels. Insuch an image, the pixel information may be processed and interpolatedso that the transition from higher to lower resolution regions occurssmoothly.

In one embodiment, zooming is achieved by inducing a barrel-likedistortion into some, or all, of the array lens so that adisproportionate number of the pixels are dedicated to the central partof each image. In this embodiment, every image has to be processed toremove the barrel distortion. To generate a wide-angle image, pixelscloser to the center are sub-sampled relative to outer pixels aresuper-sampled. As zooming is performed, the pixels at the periphery ofthe imagers are progressively discarded and the sampling of the pixelsnearer the center of the imager is increased.

In one embodiment, mipmap filters are built to allow images to berendered at a zoom scale that is between the specific zoom range of theoptical elements (e.g., 1× and 3× zoom scales of the camera array).Mipmaps are a precalculated optimized set of images that accompany abaseline image. A set of images associated with the 3× zoom luma imagecan be created from a baseline scale at 3× down to 1×. Each image inthis set is a version of the baseline 3× zoom image but at a reducedlevel of detail. Rendering an image at a desired zoom level is achievedusing the mipmap by (i) taking the image at 1× zoom, and computing thecoverage of the scene for the desired zoom level (i.e., what pixels inthe baseline image needs to be rendered at the requested scale toproduce the output image), (ii) for each pixel in the coverage set,determine if the pixel is in the image covered by the 3× zoom lumaimage, (iii) if the pixel is available in the 3× zoom luma image, thenchoose the two closest mipmap images and interpolate (using smoothingfilter) the corresponding pixels from the two mipmap images to producethe output image, and (iv) if the pixel is unavailable in the 3× zoomluma image, then choose the pixel from the baseline 1× luma image andscale up to the desired scale to produce the output pixel. By usingmipmaps, smooth optical zoom may be simulated at any point between twogiven discrete levels (i.e., 1× zoom and 3× zoom).

In one embodiment, zooming is achieved by realizing different Fields OfView (FOV)s by electronically switching between different opticalchannels having different sensor sizes, but fixed Effective FocalLengths (EFL)s. In one such embodiment, shown schematically in FIG. 8A,variable FOVs are achieved by creating optical channels on the samesubstrate that have different imager sizes 800 and 802 at the same fixedEFL 804. Using this structure, it would be possible to create anarbitrary number of zoom magnifications by including image sensors withlarger or smaller numbers of pixels. This technique is particularlysimple to incorporate into WLO array cameras as these variable zoomsensor arrays 800 and 802 can be fabricated directly onto the basecamera array substrate without any further modification to the design ofthe array camera assembly itself.

In another embodiment, as shown in FIG. 8B, different FOVs are achievedby engineering different EFLs 805 into specific optical channels of thecamera array 806 while maintaining a fixed imager size 808. Implementingdifferent EFLs on the same substrate stacks, i.e., substrate stacks withconstant thicknesses and spacings, is more complicated, because thedistance of the principal plane and with it the entrance pupil andconsequently the aperture stop 810 with respect to the image sensor 814needs to be changed in order to change the focal length of the opticalchannel. In the current embodiment, this is accomplished by theintroduction of “dummy substrates” 816, 818 and 820 into the stack 806,such that each zoom channel 822, 824 and 826 has an associated aperturestop 828, 830 and 832 disposed on a different substrate or a differentface of a substrate, such that different EFLs can be achieved. As shown,while the distribution and positioning of the lenses (834, 836 and 838)and the aperture stops (828, 830 and 832) on the particular substrate orsubstrate face is entirely dependent on the desired EFL, in all casesthe substrate thicknesses and distances remain fixed. Alternatively, insuch an embodiment, each of the substrates could be provided withlenses, but in different distributions in order to allow for differentEFLs. Such a structure would allow for higher image quality, but athigher cost.

In another further embodiment, as shown in FIG. 8C, different FOVs canalso be achieved by engineering different EFLs 805 using “dummy”substrates in a manner similar to that illustrated in FIG. 8B, with theexception that all of the substrates have lenses elements on them ineach optical channel. The lens elements, however, have differentprescriptions in order to allow different EFLs. Accordingly, any of avariety of configurations of optics and sensor size, and/or lightsensing element size can be utilized with an array camera in accordancewith embodiments of the invention to achieve different FOVs.

Capturing Video Images

In one embodiment, the camera array generates high frame imagesequences. The imagers in the camera array can operate independently tocapture images. Compared to conventional image sensors, the camera arraymay capture images at the frame rate up to N time (where N is the numberof imagers). Further, the frame period for each imager may overlap toimprove operations under low-light conditions. To increase theresolution, a subset of imagers may operate in a synchronized manner toproduce images of higher resolution. In this case, the maximum framerate is reduced by the number of imagers operating in a synchronizedmanner. The high-speed video frame rates can enables slow-motion videoplayback at a normal video rate.

In one example, two luma imagers (green imagers or near-IR imagers), twoblue imagers and two green imagers are used to obtain high-definition1080p images. Using permutations of four luma imagers (two green imagersand two near-IR imagers or three green imagers and one near-IR imager)together with one blue imager and one red imager, the chroma imagers canbe upsampled to achieve 120 frames/sec for 1080p video. For higher framerate imaging devices, the number of frame rates can be scaled uplinearly. For Standard-Definition (480p) operation, a frame rate of 240frames/sec may be achieved using the same camera array.

Conventional imaging devices with a high-resolution image sensor (e.g.,8 Megapixels) use binning or skipping to capture lower resolution images(e.g., 1080p30, 720p30 and 480p30). In binning, rows and columns in thecaptured images are interpolated in the charge, voltage or pixel domainsin order to achieve the target video resolutions while reducing thenoise. In skipping, rows and columns are skipped in order to reduce thepower consumption of the sensor. Both of these techniques result inreduced image quality.

In one embodiment, the imagers in the camera arrays are selectivelyactivated to capture a video image. For example, 9 imagers (includingone near-IR imager) may be used to obtain 1080p (1920×1080 pixels)images while 6 imagers (including one near-IR imager) may be used toobtain 720p (1280×720 pixels) images or 4 imagers (including one near-IRimager) may be used to obtain 480p (720×480 pixels) images. Becausethere is an accurate one-to-one pixel correspondence between the imagerand the target video images, the resolution achieved is higher thantraditional approaches. Further, since only a subset of the imagers isactivated to capture the images, significant power savings can also beachieved. For example, 60% reduction in power consumption is achieved in1080p and 80% of power consumption is achieved in 480p.

Using the near-IR imager to capture video images is advantageous becausethe information from the near-IR imager may be used to denoise eachvideo image. In this way, the camera arrays of embodiments exhibitexcellent low-light sensitivity and can operate in extremely low-lightconditions. In one embodiment, super-resolution processing is performedon images from multiple imagers to obtain higher resolution videoimagers. The noise-reduction characteristics of the super-resolutionprocess along with fusion of images from the near-IR imager results in avery low-noise images.

In one embodiment, high-dynamic-range (HDR) video capture is enabled byactivating more imagers. For example, in a 5×5 camera array operating in1080p video capture mode, there are only 9 cameras active. A subset ofthe 16 cameras may be overexposed and underexposed by a stop in sets oftwo or four to achieve a video output with a very high dynamic range.

Other Applications for Multiple Imagers

In one embodiment, the multiple imagers are used for estimating distanceto an object in a scene. Since information regarding the distance toeach point in an image is available in the camera array along with theextent in x and y coordinates of an image element, the size of an imageelement may be determined. Further, the absolute size and shape ofphysical items may be measured without other reference information. Forexample, a picture of a foot can be taken and the resulting informationmay be used to accurately estimate the size of an appropriate shoe.

In one embodiment, reduction in depth of field is simulated in imagescaptured by the camera array using distance information. The cameraarrays according to the present invention produce images with greatlyincreased depth of field. The long depth of field, however, may not bedesirable in some applications. In such case, a particular distance orseveral distances may be selected as the “in best focus” distance(s) forthe image and based on the distance (z) information from parallaxinformation, the image can be blurred pixel-by-pixel using, for example,a simple Gaussian blur. In one embodiment, the depth map obtained fromthe camera array is utilized to enable a tone mapping algorithm toperform the mapping using the depth information to guide the level,thereby emphasizing or exaggerating the 3D effect.

In one embodiment, apertures of different sizes are provided to obtainaperture diversity. The aperture size has a direct relationship with thedepth of field. In miniature cameras, however, the aperture is generallymade as large as possible to allow as much light to reach the cameraarray. Different imagers may receive light through apertures ofdifferent sizes. For imagers to produce a large depth of field, theaperture may be reduced whereas other imagers may have large aperturesto maximize the light received. By fusing the images from sensor imagesof different aperture sizes, images with large depth of field may beobtained without sacrificing the quality of the image.

In one embodiment, the camera array according to the present inventionrefocuses based on images captured from offsets in viewpoints. Unlike aconventional plenoptic camera, the images obtained from the camera arrayof the present invention do not suffer from the extreme loss ofresolution. The camera array according to the present invention,however, produces sparse data points for refocusing compared to theplenoptic camera. In order to overcome the sparse data points,interpolation may be performed to refocus data from the spare datapoints.

In one embodiment, each imager in the camera array has a differentcentroid. That is, the optics of each imager are designed and arrangedso that the fields of view for each imager slightly overlap but for themost part constitute distinct tiles of a larger field of view. Theimages from each of the tiles are panoramically stitched together torender a single high-resolution image.

In one embodiment, camera arrays may be formed on separate substratesand mounted on the same motherboard with spatial separation. The lenselements on each imager may be arranged so that the corner of the fieldof view slightly encompasses a line perpendicular to the substrate.Thus, if four imagers are mounted on the motherboard with each imagerrotated 90 degrees with respect to another imager, the fields of viewwill be four slightly overlapping tiles. This allows a single design ofWLO lens array and imager chip to be used to capture different tiles ofa panoramic image.

In one embodiment, one or more sets of imagers are arranged to captureimages that are stitched to produce panoramic images with overlappingfields of view while another imager or sets of imagers have a field ofview that encompasses the tiled image generated. This embodimentprovides different effective resolution for imagers with differentcharacteristics. For example, it may be desirable to have more luminanceresolution than chrominance resolution. Hence, several sets of imagersmay detect luminance with their fields of view panoramically stitched.Fewer imagers may be used to detect chrominance with the field of viewencompassing the stitched field of view of the luminance imagers.

In one embodiment, the camera array with multiple imagers is mounted ona flexible motherboard such that the motherboard can be manually bent tochange the aspect ratio of the image. For example, a set of imagers canbe mounted in a horizontal line on a flexible motherboard so that in thequiescent state of the motherboard, the fields of view of all of theimagers are approximately the same. If there are four imagers, an imagewith double the resolution of each individual imager is obtained so thatdetails in the subject image that are half the dimension of details thatcan be resolved by an individual imager. If the motherboard is bent sothat it forms part of a vertical cylinder, the imagers point outward.With a partial bend, the width of the subject image is doubled while thedetail that can be resolved is reduced because each point in the subjectimage is in the field of view of two rather than four imagers. At themaximum bend, the subject image is four times wider while the detailthat can be resolved in the subject is further reduced.

Offline Reconstruction and Processing

The images processed by the imaging system 400 may be previewed beforeor concurrently with saving of the image data on a storage medium suchas a flash device or a hard disk. In one embodiment, the images or videodata includes rich light field data sets and other useful imageinformation that were originally captured by the camera array. Othertraditional file formats could also be used. The stored images or videomay be played back or transmitted to other devices over various wired orwireless communication methods.

In one embodiment, tools are provided for users by a remote server. Theremote server may function both as a repository and an offlineprocessing engine for the images or video. Additionally, applets mashedas part of popular photo-sharing communities such as Flikr, Picasaweb,Facebook etc. may allow images to be manipulated interactively, eitherindividually or collaboratively. Further, software plug-ins into imageediting programs may be provided to process images generated by theimaging device 400 on computing devices such as desktops and laptops.

Various modules described herein may comprise a general-purpose computerselectively activated or reconfigured by a computer program stored inthe computer. Such a computer program may be stored in a computerreadable storage medium, such as, but is not limited to, any type ofdisk including floppy disks, optical disks, CD-ROMs, magnetic-opticaldisks, read-only memories (ROMs), random access memories (RAMs), EPROMs,EEPROMs, magnetic or optical cards, application specific integratedcircuits (ASICs), or any type of media suitable for storing electronicinstructions, and each coupled to a computer system bus. Furthermore,the computers referred to in the specification may include a singleprocessor or may be architectures employing multiple processor designsfor increased computing capability.

While particular embodiments and applications of the present inventionhave been illustrated and described herein, it is to be understood thatthe invention is not limited to the precise construction and componentsdisclosed herein and that various modifications, changes, and variationsmay be made in the arrangement, operation, and details of the methodsand apparatuses of the present invention without departing from thespirit and scope of the invention as it is defined in the appendedclaims.

What is claimed:
 1. A camera array, comprising: a plurality of imagers, where each imager comprises: optics comprising at least one lens element and at least one aperture that form an optical channel; an sensor comprising a two dimensional array of pixels and control circuitry for controlling imaging parameters; and at least one spectral filter located within the imager, where each spectral filter is configured to pass a specific spectral band of light; wherein the plurality of imagers are configured to capture images of the same scene; wherein the shape of at least one lens element of an imager of a first type that includes a spectral filter configured to pass a first spectral band of light differs from the shape of at least one lens element forming a second optical channel for an imager of a second type that includes a spectral filter configured to pass a second spectral band of light; wherein the images captured by the plurality of imagers include different occlusions sets, where the occlusion set of a given imager is the portion of a scene visible to a baseline imager in the plurality of imagers that is occluded from the view of the given imager; and wherein the plurality of imagers includes a first imager that captures pixels around an edge of a foreground object that is visible to the baseline imager and is in the occlusion set of a second of the plurality of imagers that has the same imaging characteristics as the first imager. 